PIPES: A Multi-Threaded Publish-Subscribe Architecture for Continuous Queries over Streaming Data Sources

In contrast to traditional query processing based on persistent data, new application scenarios arise that heavily rely on the continuous evaluation of data streams. These streams often emerge from autonomous data sources. In this paper, we present the novel publish-subscribe architecture PIPES (Public Infrastructure for Processing and Exploring Streams), which easily allows the composition of complex query graphs with an inherently dynamic resource sharing, even during runtime. We sketch essential design and implementation considerations of PIPES, which contains a generic operator framework for streaming data. PIPES is completely implemented and integrated in XXL, a flexible and extensible Java library for data processing. In contrast to related system prototypes, whose communication between operators solely depends on queues, we permit direct interoperability. In our new hybrid scheduling approach for PIPES, we intend to dynamically vary the number of concurrently running lightweight processes, called threads. First experimental studies show its advantages of neither being restricted to a single thread nor assigning a separate thread per operator.